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Executive  Summary 


This  document  reports  the  results  of  the  evaluation  of  the  National  Airspace  System  (NAS) 
Infrastructure  Management  System  (NIMS)  Build  1 .5  computer-human  interface  (CHI)  software. 
Human  factors  engineers  from  the  Federal  Aviation  Administration  (FAA)  William  J.  Hughes 
Technical  Center  NAS  Human  Factors  Branch  (ACT-530)  conducted  the  evaluation  at  the  NIMS 
Premier  Facility  (NPF)  in  Herndon,  Virginia  from  November  16  to  19, 1998.  Five  Airway 
Facilities  (AF)  Specialists,  serving  as  representative  users,  and  three  human  factors  engineers, 
serving  as  expert  observers,  participated  in  the  evaluation. 

The  goal  of  the  evaluation  was  to  identify  CHI  issues  prior  to  completing  software  development. 
The  evaluation  employed  a  structured  walk-through  technique  with  the  users  providing 
comments  as  they  performed  test  scripts.  These  test  scripts  exercised  the  three  software 
applications  that  compose  NIMS  Build  1.5.  In  parallel,  the  observers  provided  relevant  human 
factors  observations. 

ACT-530  staff  organized  and  analyzed  the  comments  obtained  from  the  users  and  observers. 

The  results  of  the  evaluation  identified  several  issues  that  could  improve  system  functionality. 
Four  major  issues  include 

a.  integrating  the  three  software  components, 

b.  providing  the  correct  information  needed  to  support  task  performance, 

c.  handling  error  messages  and  error  management,  and 

d.  providing  proper  and  sufficient  feedback  regarding  user  and  system  actions. 

The  AF  Specialists  in  the  study  indicated  that  at  its  current  level  of  maturity,  NIMS  Build  1 .5 
was  not  capable  of  providing  this  support  in  an  operational  environment.  However,  they 
anticipated  that  if  the  issues  they  identified  were  addressed,  a  future  build  of  NIMS  had  the 
potential  to  support  a  centralized  monitoring  and  control  facility.  The  human  factors  engineers 
concluded  that  the  majority  of  issues  identified  could  be  mitigated  through  the  application  of 
human  factors  principles. 
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1.  Introduction 


At  the  request  of  the  National  Airspace  System  (NAS)  Infrastructure  Management  System 
(NIMS)  Product  Team,  human  factors  engineers  from  the  NAS  Human  Factors  Branch  (ACT- 
530)  of  the  Federal  Aviation  Administration  (FAA)  William  J.  Hughes  Technical  Center 
conducted  an  evaluation  of  NIMS  Build  1.5  software.  This  report  summarizes  the  results  of  the 
usability  evaluation  conducted  in  conjunction  with  the  NAS  Operations  NIMS  Program  Office 
(AOP-30)  during  the  week  of  November  16, 1998.  It  identifies  and  systematically  classifies  the 
human  factors  issues  identified  regarding  the  Build  1.5  NIMS  computer-human  interface  (CHI). 

The  evaluation  was  conducted  at  the  NIMS  Premier  Facility  (NPF)  in  Herndon,  Virginia.  The 
NPF  serves  as  the  site  to  consider  the  viability  of  NIMS  functionality  in  terms  of  user  acceptance 
and  productivity  (FAA,  1998b).  This  evaluation  reviewed  the  NIMS  Specialist  interface  in  terms 
of  usability  and  applicable  CHI  standards  contained  in  the  Airway  Facilities  (AF)  Human  Factors 
Design  Guide  (HFDG)  (Wagner,  Birt,  Snyder,  &  Duncanson,  1996)  and  other  relevant  human 
factors  sources. 

1.1  Background 

The  goal  of  the  NIMS  Program  was  to  transition  from  a  decentralized  organization  focused  on 
equipment  maintenance  to  a  more  centralized  organization  focusing  on  service  management, 
prioritizing  actions  on  the  basis  of  operational  significance,  and  improving  service  to  users  and 
customers.  The  NIMS  Program  leveraged  new  technologies  to  enhance  the  efficiency  and 
effectiveness  of  NAS  infrastructure  management.  Specifically,  it  implemented  a  new  approach 
focused  on  the  principles  of  service  management,  nationwide  operations  planning  and  oversight, 
centralized  operational  control,  consolidated  expertise,  remote  management  of  equipment, 
integrated  information  infrastructure,  and  collaborative  decision-making.  The  NAS 
infrastructure  operations  and  maintenance  concept  calls  for  establishing  an  operations 
environment  consisting  of  one  National  Operations  Control  Center,  three  Operations  Control 
Centers,  and  a  larger  number  of  Work  Centers  (FAA,  1997).  The  NIMS  Program  was  intended 
to  provide  automation  support  services  for  the  management  of  the  NAS  infrastructure  (FAA, 
1998b). 

1 .2  Scope  and  Limitations 

This  evaluation  was  limited  to  an  assessment  of  the  usability  of  Build  1.5  of  the  NIMS  Specialist 
interface.  Build  1.5  was  a  selected  subset  of  several  capabilities,  which  the  FAA  compiled  from 
the  NIMS  System/Segment  Specification  (FAA,  1998c).  It  represented  a  subset  of  functionality  to 
be  contained  in  NIMS  Build  2.  This  report  focuses  on  the  ability  of  the  user  interface  to  support 
the  specialist  tasks.  Data  on  the  safety,  efficiency,  performance,  and  workload  levels  associated 
with  this  design  were  not  collected.  These  data  represent  the  type  of  evaluations  that  may  be 
conducted  when  the  system  nears  maturity. 
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The  following  list  itemizes  the  specific  limitations  to  this  evaluation: 

a.  The  overall  NIMS  software  involved  three  major  applications:  Maximo,  Maintenance 
Automation  System  Software  (MASS),  and  ArcView.  All  came  from  different  sources 
and  only  minimal  integration  had  been  accomplished. 

b.  The  Maximo  software  was  not  in  its  final  form.  Several  modifications  to  the  commercial 
product,  which  were  required  to  meet  final  FAA  requirements,  were  not  yet 
accomplished. 

c.  The  system  server  was  a  smaller,  more  limited  one  than  planned  for  operational  use. 

d.  A  significant  software  element,  the  Security  and  User  Access  module,  was  not  in  use. 

e.  The  absence  of  some  planned  components  such  as  on-line  help  and  on-line  technical 
manuals  made  evaluation  of  related  operations  impossible. 

f.  The  database  used  was  minimally  populated. 

g.  The  users  and  observers  received  minimal  training  on  the  system  prior  to  the  evaluation. 
2.  Method 

This  evaluation  employed  a  methodology  that  ACT-530  had  developed  and  refined  to  determine 
the  CHI  usability  of  FAA  systems.  This  was  a  very  applied  environment  with  restrictions  as 
listed  above.  As  requested,  human  factors  specialists  worked  with  what  they  had  in  terms  of 
system  availability. 

2.1  Participants 

Two  groups  participated  in  the  evaluation  simultaneously.  The  first  group  consisted  of  five 
representative  users  recruited  by  AOP-30.  The  second  group  consisted  of  human  factors 
engineers  from  ACT-530  who  provided  human  factors  observations.  Table  1  provides  the 
information  for  both  groups. 


Table  1 .  Participant  Background  Information 


Position 

Region 

Systems  Specialist 

Southwest 

Systems  Specialist 

Western-Pacific 

Systems  Specialist 

Southern 

NAS  Operations  Manager 

Northwest  Mountain 

NAS  Operations  Manager 

Great  Lakes 

Human  Factors  Engineers  (3) 

Technical  Center 

2 


2.2  Materials 


The  equipment  used  to  conduct  the  evaluation  consisted  of  hardware  and  software  located  at  the 
NPF  and  the  video  capture  equipment  supplied  by  ACT-530.  Table  2  identifies  this  equipment. 

Table  2.  Test  Equipment 


Hardware 

Software 

OCC  workstations  each  consisting  of 

Software  loaded  on  each  OCC  workstation 

•  Pentium  PC 

•  Windows  NT  Server  4.0  (Service  package  3) 

•  128  MB  RAM 

•  Oracle  Client  7.3.2.2.0 

•  2  GB  Hard  Drive 

•  MAXIMO  version  4.0  Build  1.5 

•  3COM  10/100  Ethernet,  PCI  Adapter 

•  MASS  version  1.0.041 

•  17”  Color  Monitor 

•  ArcView 

•  104  Key  Keyboard 

•  Microsoft  Office  97 

•  12/24X  SCSI  CD-ROM 

•  Norton  AntiVirus 

•  1 .44  MB  Floppy  Drive 

Screen  capture  equipment 

•  Scan  Converter 

•  Video  Recorder 

•  Microphone 

ACT-530,  working  in  conjunction  with  AOP-30,  developed  six  test  scripts  (Appendix  A)  to 
exercise  selected  functionality  present  in  the  Build  1.5  NIMS  Specialist  interface.  ACT-530  and 
AOP-30  developed  them  from  procedures  contained  in  the  NIMS  NPF  Integration  Test  Plan  and 
Procedures  (FAA,  1 998a)  and  other  relevant  materials.  The  first  three  scripts  stepped 
participants  through  the  functionality  of  MASS,  ArcView,  and  MAXIMO.  The  remaining  three 
scripts  represented  end-to-end  tasks  using  a  combination  of  all  three  applications.  Script  four 
addressed  alarm  detection,  Candidate  Event  Ticket  (CET)  generation,  and  Event  Ticket  (ET) 
generation.  Scripts  five  and  six  walked  the  participants  through  the  procedure  to  build  new  ETs 
and  create  and  track  workforce  schedules. 
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2.3  Schedule 


Table  3  provides  the  schedule  for  this  evaluation. 

Table  3.  NIMS  Build  1.5  CHI  Evaluation  Schedule 


Time 

November  17 

November  18 

November  19 

Morning 

NIMS  Build  1.5  orientation 

Human  factors  overview 

Self-paced  walk-through  of 
scripts 

Self-paced  walk-through  of 
scripts 

Observation  of  interactions 

Self-paced  walk-through  of 
scripts 

Observation  of  interactions 

Usability  questionnaire  i 

Afternoon 

Self-paced  walk-through  of 
scripts 

Observation  of  interactions 

Self-paced  walk-through  of 
scripts 

Observation  of  interactions 

Debrief 

2.4  Procedure 


This  evaluation  employed  a  structured  walk-through  technique.  The  method  consisted  of  users 
following  scripts  exercising  common  tasks  and  providing  comments.  In  parallel,  the  human 
factors  observers  reviewed  the  interface  against  accepted  human  factors  practices.  The  users 
completed  a  Post-Usability  Evaluation  Questionnaire  (Appendix  B)  at  the  conclusion  of  all  test 
scripts.  The  questionnaire  assessed  specific  human  factors  aspects  of  the  specialist  interface 
including  screen  design,  data  entry,  error  messages,  and  input  and  control  devices.  The  observers 
used  a  scan  converter  and  video  recorder  to  capture  screen  activity  on  a  selected  workstation 
while  a  participant  completed  the  scripts.  The  observers  reviewed  this  videotape  during  the  data 
analysis  activities. 

The  users  performed  the  scripts  independently  on  each  of  the  five  available  OCC  workstations  at 
the  NPF.  At  the  conclusion  of  all  test  scripts,  they  completed  the  questionnaire.  The  observers 
reviewed  the  NIMS  Build  1 .5  Specialist  interface  while  users  performed  the  test  scripts.  This 
review  included  an  evaluation  of  the  interface  with  respect  to  human  factors  guidelines  contained 
in  the  HFDG  (Wagner  et  al.,  1996)  and  other  relevant  sources. 

At  the  conclusion  of  the  study,  the  observers  integrated  all  comments  and  questionnaire 
responses  and  performed  a  content  analysis.  They  compiled  and  categorized  the  resulting  list  of 
items  into  relevant  human  factors  issues.  These  are  presented  in  the  following  sections. 
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3.  Results  and  Discussion 


The  participants  provided  350  comments.  The  human  factors  engineers  integrated  and 
categorized  the  comments  and  conducted  a  human  factors  guideline  review.  They  classified  the 
issues  into  the  following  categories:  general,  basic  screen  design  and  operation,  data  entry,  data 
display,  user  access,  and  help. 

3.1  General 


Most  users  indicated  on  the  questionnaire  that  the  system  effectively  supported  their  ability  to 
manage  resources,  communicate,  and  generate  reports.  Two  rated  the  system  as  adequate  in 
terms  of  supporting  coordination.  However,  all  agreed  that  the  current  design  was  not 
acceptable.  It  required  too  much  interaction  and  typing  and  had  too  many  software  problems. 
Though  they  considered  the  system  to  be  very  capable,  they  felt  it  would  require  significant 
changes  before  it  could  be  fielded.  Overall,  the  users  were  not  satisfied  with  any  of  the 
functionality  in  its  current  state.  Some  users  acknowledged  that  their  review  was  hampered  by 
the  lack  of  a  fully  populated  database  and  recommended  that  a  complete  national  database  be 
available  for  future  evaluations. 

The  users  commented  on  the  lack  of  integration  between  the  three  applications.  For  example, 
when  a  loss  of  redundancy  or  other  event  was  represented  in  ArcView,  they  found  it  cumbersome 
to  identify  the  corresponding  ET  in  the  Maximo  CET  database.  They  were  required  to  scroll 
through  a  long  list  of  CETs  to  locate  the  appropriate  one.  When  required  to  review  or  take 
control  of  the  data  stream  from  that  facility,  they  had  to  open  MASS  and  filter  through  all 
national  facilities  to  locate  the  appropriate  selection.  As  another  example,  when  MASS  alarms 
were  acknowledged  and  notes  were  made,  these  notes  were  not  available  to  Maximo. 

In  addition  to  these  broad  issues,  the  users  and  the  observers  commented  on  other  aspects  of  the 
system  such  as  input  devices,  response  time,  feedback,  error  messages,  database  population  and 
maintenance,  and  software  bugs.  These  are  addressed  below. 

3.1.1  Input  Devices 

The  users  indicated  that  the  keyboard  and  mouse  worked  well  for  all  tasks  (including  data  entry 
and  cursor  positioning).  As  an  example,  Maximo  accepted  both  keyboard  and  mouse  inputs 
interchangeably  in  date  and  time  fields. 

3. 1 .2  System  Response  Time 

The  NPF  server  used  during  this  evaluation  was  not  representative  of  the  anticipated  final 
configuration.  The  user  comments  regarding  system  response  times  in  the  following  paragraph 
are  provided  to  highlight  the  need  for  adequate  server  response  times  in  future  systems.  All  users 
indicated  that  the  system  response  times  were  not  acceptable. 
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Initial  load  times  for  ArcView,  a  graphics-intensive  application,  required  approximately  15 
minutes,  1  Vi  minutes  to  add  a  group,  and  up  to  several  seconds  to  refresh  a  screen.  Most  users 
indicated  that  selecting  the  terminal,  radar,  or  remote  options  from  the  ArcView  drop  down  list 
(step  2.  If  of  the  script)  was  especially  slow,  taking  an  estimated  2  minutes  or  more  to  load  for 
each  option.  Initial  load  times  for  Maximo  screens  ranged  from  1  to  3  minutes,  and  subtabs  took 
approximately  12  seconds  to  load.  One  user  reported  that  Maximo  refresh  rates,  after  initial 
loading,  were  acceptable.  Some  users  found  MASS  response  times  to  be  slow.  Observation  of 
the  videotapes  indicated  that  logging  on  to  the  MASS  system  took  approximately  12  seconds  and 
that  screen  updates  took  about  2  seconds. 

3.1.3  Feedback 

The  user  did  not  always  receive  feedback  on  the  status  of  a  process.  On  the  questionnaire,  all 
users  indicated  that  they  were  often  unaware  of  what  the  system  was  doing.  The  HFDG  (Wagner 
et  al.,  1996)  recommends  that  when  system  responses  take  more  than  2  seconds,  the  user  should 
be  provided  periodic  feedback  and  a  clear  indication  of  when  processing  is  complete.  In  many 
instances,  ArcView  and  Maximo  did  not  adequately  provide  this  type  of  feedback.  Users  noted 
the  following  examples: 

a.  It  took  several  minutes  to  load  ArcView. 

b.  Users  received  no  feedback  regarding  the  loading  status  and,  in  many  cases,  had  to 
minimize  and  remaximize  the  window  at  the  completion  of  loading  to  begin  using  the 
application. 

c.  A  user  can  open  a  new  instance  of  ArcView  without  being  alerted  that  they  already  have  a 
copy  opened. 

d.  When  logging  on  to  ArcView,  the  user  is  not  notified  that  the  password  is  case  sensitive. 
This  should  be  stated  in  the  box  where  the  password  is  entered  or  provided  in  response  to 
a  failed  logon  attempt. 

e.  When  ArcView  is  minimized,  there  is  no  indication  of  alarm  status  or  changes  in  alarm 
status. 

f.  Maximo  did  not  notify  the  users  when  they  were  at  the  first  ET  and  selected  the  “view 
previous  ticket”  icon. 

g.  Maximo  does  not  always  provide  feedback  when  loading  screens. 

h.  Maximo  does  not  notify  a  user  when  an  open  ET  is  modified.  Remote  AF  System 
Specialists  and  other  personnel  will  frequently  access  and  update  ETs.  Without  some 
form  of  notification,  the  OCC  user  is  unaware  of  which  open  ETs  need  to  be  refreshed.  If 
the  OCC  user  fails  to  update  a  modified  ET,  the  change  may  go  unnoticed  for  an  extended 
period  and  have  an  impact  on  the  delivery  of  NAS  services. 
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3.1.4  Error  Messages 


Error  messages  hold  significant  implications  for  user  performance.  Shneiderman  (1992)  notes 
that  in  a  review  of  five  studies,  improved  error  messages  resulted  in  lower  error  rates,  increased 
user  satisfaction,  and  more  success  in  repairing  errors. 

All  of  the  participants  indicated  that  error  messages  were  not  clear  and  that  the  messages  did  not 
summarize  the  problem  or  provide  solutions.  The  messages  referenced  variables  that  were 
unfamiliar  to  the  users  and  did  not  instruct  them  in  how  to  recover  from  the  error.  The  error 
messages  were  not  given  in  language  that  the  user  could  understand  and  did  not  give  a  clear 
indication  of  possible  solutions.  Examples  of  some  error  messages  are  provided. 

a.  When  attempting  to  log  on  to  ArcView,  the  system  indicated  Variable  cover  Ftab  has  not 
been  initiated. 

b.  In  response  to  selecting  themes  in  ArcView,  an  error  message  appeared  sidXmgAvenne 
Error:  NFMD.  Display  Adjacent  Radar  Tool  <Stop>  Variable  cover  Ftab  has  not  been 
initiated. 

c.  When  closing  the  ArcView  location  table,  the  following  message  was  received — Error. 
The  script  ‘‘NFMD,  getAppLoglndex ”  could  not  be  found. 

d.  In  some  instances,  when  a  user  was  attempting  to  change  the  status  of  an  ET  in  Maximo 
(Step  4.4j),  the  system  indicated  SQL  Error:  904-ORA-00904:  Invalid  Column  Name. 
When  they  selected  “more”  to  get  additional  information,  they  received  the  following 
sequel  (SQL)  statement:  select  WONUM from  WOsafetylink  where  NONUM  =  :jwonum 
and  wosafetydatasoure=  ’WO ’. 

e.  When  creating  a  Significant  Event  Report  (SER)  in  Maximo  (step  4.6e  of  the  scripts),  one 
user  was  informed  that  ET  1558  was  not  a  significant  event.  The  user  was  unsure  how  to 
proceed. 

f.  Another  error  message  received  in  response  to  an  attempt  to  print  an  SER  stated  Report 
not  generated. 

Many  of  the  errors  resulted  in  the  need  to  reload  the  application  or  to  reboot  the  workstation. 

The  human  factors  engineers  recommend  that  error  messages  follow  the  guidelines  provided  in 
the  HFDG  (Wagner  et  al.,  1996)  and  that  they  are  brief,  informative,  and  provide  specific 
direction  to  users  on  how  to  recover  from  an  error.  If  required,  the  system  may  provide  more 
than  one  level  of  error  messages  with  successive  levels  providing  more  detail. 

3.1.5  Database  Population  and  Maintenance 

The  users  acknowledged  the  value  of  many  of  the  types  of  data  available  in  the  system. 

However,  their  major  concern  was  identifying  who  would  be  responsible  for  entering  and 
maintaining  the  data.  The  tasks  of  entering  and  updating  the  databases  are  extremely  time 
intensive  and  require  extensive  manpower.  One  user  identified  the  calendar  as  a  particularly 
good  tool  but  asked  who  would  maintain  the  technician  schedule  and  update  overtime. 
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This  concern  may  have  been  raised  in  part  because  the  users  represented  potential  OCC  operators 
and  many  of  the  screens  they  viewed  (e.g.,  accounting  screens)  represented  information  that  they 
would  most  likely  not  use  in  an  operational  system.  The  human  factors  engineers  recommend 
that  a  detailed  review  of  the  interface  be  conducted  with  other  groups  of  representative  users 
(e.g.,  administrative,  training,  analysis,  and  management)  to  identify  specific  screen  content  and 
format  needs  for  each  type  of  user. 

3.1.6  Software  Bugs 

Although  not  the  primary  focus,  during  the  evaluation,  the  participants  identified  several  software 
bugs  in  the  Maximo  and  ArcView  applications: 

a.  On  several  occasions,  when  a  user  attempted  to  run  a  report  or  change  the  status  of  an 
action,  Maximo  returned  the  SQL  error  discussed  previously.  The  source  of  this  error 
message  was  not  clear  to  the  participants. 

b.  Selecting  the  help  button  when  viewing  the  ET  screen  in  Maximo  resulted  in  the  shut 
down  of  the  application  and  the  generation  of  an  event  log. 

c.  When  running  a  report  in  Maximo  (Step  4.6f  of  the  script),  values  entered  in  the  delay 
field  screen  were  doubled. 

d.  In  Maximo,  when  completing  the  ticket  priority  field  of  an  ET,  the  default  value  is  5,  but 
the  available  options  are  1,  2,  or  3. 

e.  When  selecting  the  terminal  drop  down  list  in  ArcView,  one  user  received  an  error 
indicating  that  the  variable  cover  Ftab  had  not  been  initiated. 

3.2  Basic  Screen  Design  and  Operations 

The  basic  principle  of  screen  design  and  operation  should  be  to  support  the  performance  of 
required  tasks  in  the  most  efficient  and  effective  manner.  It  includes  the  overall  design,  structure, 
and  content  of  the  individual  screens;  how  the  screens  are  used;  and  how  users  select  and 
navigate  among  them. 

Each  of  the  following  subsections  addresses  a  particular  aspect  of  screen  design  or  operation.  In 
all,  there  were  192  comments  assigned  to  this  category,  and  these  were  sorted  into  nine  primary 
subcategories.  Many  of  the  issues  are  interrelated,  so  the  comments  may  address  more  than  one 
subcategory. 

3.2.1  Consistency 

A  basic  concept  of  CHI  design  is  that  any  task,  operation,  or  action  should  always  be 
accomplished  the  same  way  throughout  the  system  (Wagner  et  al.,  1996,  Sections  5.1.1  and 
8.5.1).  This  helps  minimize  operator  error,  training  requirements,  and  task  loading.  Changes 
should  only  occur  when  there  is  a  substantive  need  to  modify  the  item  or  object.  Users  should 
not  have  to  remember  multiple  identifiers  or  names  for  a  single  object.  Where  a  new  system 
design  must  interface  with  existing  (legacy)  systems,  an  attempt  should  be  made  to  be  consistent 
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with  current  practice  unless  significant  changes  in  process  or  content  require  differentiation  from 
the  other  systems. 

Specialists  and  observers  made  several  comments  regarding  consistency.  Some  of  these  were 
also  relevant  to  other  issues. 

a.  Capitalization  during  data  entry  was  inconsistent.  Some  fields  were  case  sensitive.  The 
user  had  no  way  of  knowing  this  except  by  memorization.  In  some  cases,  the  user  was 
unable  to  determine  why  the  entry  was  not  accepted  until  informed  by  someone  else. 

b.  Each  CET  generated  by  MASS  had  an  identification  number.  When  accepted  by  the 
specialist  as  an  ET,  a  different  number  was  assigned.  This  made  it  difficult  to  trace  the 
CET/ET  history  and  relationship  or  to  find  out  whether  a  given  CET  has  been  accepted  or 
cancelled.  This  problem  received  multiple  comments. 

c.  Use  of  the  percent  sign  (%)  as  a  wild  card  during  text  searches  was  available  for  some 
fields  but  not  others.  The  user  had  no  way  of  knowing  whether  a  search  failed  because 
the  field  didn’t  accept  the  %  symbol  or  for  some  other  reason.  The  alternative  is  a  global 
search,  which  may  be  slower  and  less  productive. 

d.  Maximizing  and  minimizing  the  MASS  window  was  performed  differently  than  the  other 
two  software  components. 

e.  Highlighting  of  fields  when  selected  was  not  consistent. 

These  items  can  make  system  operation  more  difficult  for  a  user.  The  human  factors  engineers 
recommend  a  review  of  the  overall  system  software  integration  to  determine  the  feasibility  of 
modifications  to  promote  internal  consistency.  Among  the  areas  that  should  be  considered  are 
sensitivity  to  capitalization  of  data  entry,  event  numbering,  use  of  wild  cards,  window-resizing 
function,  and  highlighting. 

3.2.2  Log  On  and  Log  Off 

Log  on  and  log  off  should  be  simple  but  precise.  Where  user  ID  is  required,  the  process  should 
provide  good  identification  without  extensive  logging  procedures  (Wagner  et  al.,  1996,  Section 
1 1.3).  Where  multiple  applications  are  involved,  additional  effort  should  be  made  to  minimize 
the  number  of  separate  log  ons  required  when  a  user  consistently  uses  the  same  set  of  _ 
applications.  Because  Build  1.5  uses  three  separate  application  packages  and  requires  three 
separate  log  ons,  this  is  of  particular  interest.  Participants  commented  on  the  long  loading  times 
for  ArcView  and  MASS  and  lack  of  feedback  to  the  user  as  to  loading  status.  The  major  issues 
raised  regarding  log  on  and  log  off  were  the  following: 

a.  It  was  possible  to  accidentally  log  off  from  some  processes  that  were  running  in  the 
background  without  the  user  being  informed. 

b.  Users  had  to  log  on  to  multiple  programs  with  different  passwords. 

The  human  factors  engineers  recommend  implementing  a  single,  system-wide  log  on. 
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3.2.3  Simplicity 


Screens  and  other  displays  should  not  contain  extraneous,  rarely  used,  or  irrelevant  data. 
Irrelevant  data  require  the  user  to  scan  the  material  to  select  the  relevant  content.  This  slows 
performance  and  can  generate  errors.  It  can  also  annoy  users  and  prevent  their  acceptance  of  the 
system. 

Almost  a  third  of  all  user  comments  addressed  the  presence  of  irrelevant  content  on  the  screens. 
It  represented  a  major  concern  in  this  build.  Additionally,  many  data  tables  were  large  and 
required  extensive  scrolling  both  vertically  and  horizontally.  This  slowed  task  performance  and 
could  generate  errors.  Examples  of  these  issues  include  the  following  items: 

a.  The  tabs  associated  with  the  ET  tracking  screens  contained  extensive  costing  data  never 
used  while  processing  an  event. 

b.  The  equipment  location,  ET  list,  and  the  MASS  alarm  tables  provided  inadequate 
filtering  and  search  support. 

This  is  a  very  significant  operational  problem.  It  clearly  concerned  the  users,  who  recommended 
deletion  of  entire  tabs,  tables,  and  screens.  The  human  factors  engineers  recommend  that  the 
screens  and  their  content  be  tailored  to  support  specific  tasks  and  users.  Not  all  personnel  will 
perform  all  tasks.  Therefore,  providing  an  individual  with  unnecessary  data  adds  needless 
complexity. 

3.2.4  Labels.  Titles.  Wording,  Borders,  and  Menus 

Use  of  labels  and  titles  should  be  consistent  throughout  a  system  (Wagner  et  al.,  1996,  Section 
8.5.2).  In  NIMS,  where  commercial-off-the-shelf  (COTS)  software  from  three  different  vendors 
is  being  integrated,  this  becomes  a  significant  issue.  Wording  used  in  the  labels  and  titles  should 
conform  to  the  uses  of  the  technical  field  involved.  For  example,  in  one  case,  an  Event  Ticket 
(FAA  term)  was  also  referred  to  as  a  Work  Order  (common  industrial  term).  An  additional  issue 
is  that  labels  must  be  informative. 

Menus  are  often  the  easiest  means  of  accessing  specific  content.  To  be  effective,  menus  must  be 
limited  in  number  and  size  and  must  have  keyboard  equivalents  where  appropriate.  Correct 
wording  and  ease  and  speed  of  access  are  significant  in  menu  design.  The  users  and  observers 
raised  no  major  issues  about  menus.  The  relatively  small  number  of  comments  were  primarily 
related  to  Maximo  and  addressed 

a.  the  presence  of  incorrect,  confusing,  or  non-informative  labels  (these  labels  do  not 
properly  describe  the  field  or  content); 

b.  the  use  of  abbreviations  for  location  identifiers  (because  the  sites  monitored  by  an  OCC 
will  be  geographically  dispersed,  specialists  would  need  to  look  up  or  memorize  the 
identifiers); 

c.  the  use  of  non-FAA  terminology  such  as  references  to  ET  components  as  “children;”  and 
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d.  the  use  of  the  same  label  for  different  content  on  many  subsidiary  screens.  Most  notably, 
the  “drilldown”  and  “select  value”  titles  are  used  for  several  very  different  content 
screens.  It  can  result  in  users  thinking  they  are  in  one  screen  when  they  are  actually  in 
another.  This  can  generate  errors. 

We  recommend  reviewing  the  use  of  labels  and  abbreviations,  particularly  within  Maximo,  to 
verify  they  are  used  consistently  and  employ  FAA  terminology. 

3.2.5  Window  Controls 


This  subcategory  addresses  the  use  of  scroll  bars,  up/down  arrows,  paging,  pointers,  and  other 
means  of  navigating  around  a  screen  when  all  content  cannot  be  displayed  at  one  time.  This 
issue  received  little  direct  comment,  with  the  exception  of  the  excessive  need  to  use  these 
controls,  which  is  addressed  in  the  Simplicity  section. 

This  issue  does  not  appear  to  have  been  a  problem  for  the  users  or  observers.  Some  adjustments 
would  probably  be  helpful  after  other  problems  have  been  resolved. 

3.2.6  Window  States 


ET  windows  can  be  in  different  operational  states.  This  affects  the  ability  of  the  user  to  enter, 
delete,  or  modify  screen  data.  Maximo  screens  have  four  possible  modes  that  affect  data  entry 
and  use.  The  only  indication  of  which  mode  is  invoked  is  a  small  block  at  the  lower  left  of  the 
screen  on  the  status  bar.  Due  to  the  minimal  training,  users  were  largely  unaware  of  these  modes 
until  attempts  to  perform  some  action  incompatible  with  the  mode  failed.  No  user  comments 
were  received  on  this  issue  despite  observers  having  noted  a  number  of  mode-related  errors. 

The  human  factors  engineers  recommend  that  users  be  made  aware  of  what  state  or  mode  is 
active  on  the  system  and  what  each  mode  implies  for  operations.  This  issue  is  closely  related  to 
feedback  and  error  management  issues. 

3.2.7  Window  Navigation 

Well-designed  systems  provide  information  for  a  task  on  a  single  screen  and  minimize  the  need 
to  navigate  between  windows.  In  NIMS,  where  there  are  three  major  applications,  good  window 
navigation  is  essential.  Navigation  can  be  performed  in  several  ways,  commonly  using  menus  or 
icons.  Many  guidelines  for  navigation  have  been  established,  which  are  included  in  Section  8.3 
of  the  HFDG  (Wagner  et  al.,  1996).  The  users  experienced  little  difficulty  with  navigation 
techniques.  One  issue  was  the  need  for  extensive  scrolling  and  the  use  of  nested  scroll  bars. 

This  made  it  easy  to  move  the  wrong  one,  unnecessarily  displacing  the  data.  Related  issues 
include  extensive  scrolling  and  data  transfer  among  screens. 

The  navigation  techniques  themselves  did  not  pose  a  major  problem.  However,  the  amount  of 
navigation  required  was  significant  and  appeared  to  slow  task  performance.  This  is  also  related 
to  screen  content. 
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3.2.8  Coding 


Coding  is  the  use  of  symbols  or  other  display  characteristics  to  uniquely  display  data  values, 
modify  display  elements,  or  as  a  “shorthand”  means  of  displaying  complex  data.  In  graphic 
displays,  shape,  size,  color,  and  motion  are  common  coding  dimensions.  For  text  presentations, 
color,  shape  (italic,  bold,  etc.),  and  size  are  also  used.  Coding  guidelines  are  available  in  Section 
8.5.4  of  the  HFDG  (Wagner  et  al.,  1996). 

Although  coding  was  not  consistent  across  the  graphic  and  text  applications,  users  were  able  to 
adapt  to  differences  between  the  applications.  It  is  unclear  whether  they  would  adapt  as 
efficiently  when  faced  with  heavy  operational  loads.  The  few  comments  received  addressed  the 
ability  to  discriminate  between  different  levels  of  coding  or  to  detect  changes  in  coding.  The 
major  color-coding  issue  was  user  difficulty  in  discriminating  between  some  colors  that  were  too 
similar  or  too  faint  to  be  easily  differentiated. 

In  the  text  applications,  the  coding  was  relatively  simple  and  largely  limited  to  the  color  of  the 
text  or  field.  MASS  also  uses  few  symbolic  codes.  ArcView  drew  the  majority  of  comments 
because  much  of  its  content  is  symbolic,  and  coding  is  used  extensively.  Overall,  coding  was 
acceptable.  However,  the  system  lacked  consistency  across  applications  and  should  be  reviewed 
once  screen  content  has  been  revised. 

3.2.9  Matching  System  to  User  Needs 

This  section  addresses  whether  the  system  provides  the  users  with  the  information  required  to 
perform  their  tasks.  If  a  user  must  search  for  information  or  make  decisions  and  act  without 
needed  information,  performance  will  be  degraded.  Good  screen  design  demands  a  good 
understanding  of  the  tasks  being  performed  by  the  user  both  in  terms  of  content  and  process. 
Besides  screen  design,  navigation,  coding,  labeling,  and  many  other  factors  also  affect  the 
availability  and  usability  of  information.  More  than  half  of  the  nearly  200  comments  received  on 
screen  design  and  operation  centered  on  matching  the  system  to  user  needs.  The  following  is  a 
summary  of  major  comments: 

a.  The  presence  of  very  large  tables  of  data  made  it  difficult  for  users  to  find  needed 
information.  In  particular,  comments  centered  on  the  data  tables  contained  in  the  MASS 
alarm  and  alert  table.  These  were  organized  by  time,  though  some  specialists  needed  it 
organized  by  site.  Users  had  to  search  the  entire  list  to  find  all  the  alerts  and  alarms  at  a 
site.  They  raised  similar  issues  regarding  the  equipment  tables  and  CET/ET  lists. 

b.  When  filtering  was  available,  it  was  inconsistent  and  insufficient.  The  users  reported  that 
the  number  of  steps  required  in  MASS  and  Maximo  was  extensive  and  time  intensive. 

c.  The  users  were  unable  to  see  changes  in  document  or  system  data  without  refreshing  the 
screens  in  Maximo.  When  data  in  the  underlying  database  changed,  the  system  did  not 
notify  the  user.  Any  updates  were  not  reflected  unless  the  user  refreshed  the  screen. 
Because  these  changes  occurred  on  a  random  schedule,  the  user  had  to  either  repeatedly 
update  the  screen  or  receive  notification  of  the  change  in  some  other  way.  They 
identified  this  as  an  issue  particularly  with  respect  to  ET  and  equipment  status. 
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d.  Needed  information  was  not  easily  accessible  in  Maximo.  For  example,  the  system  did 
not  provide  a  means  for  easily  monitoring  such  variables  as  status  of  coordination 
approvals.  The  ET  form  did  not  provide  information  on  whether  approvals  had  been 
obtained  or  denied,  and  the  user  had  to  locate  the  information  in  other  areas  of  the  system. 

e.  Needed  data  were  not  available  in  ArcView.  Some  data  considered  important  by  the 
users  were  not  present,  including  information  on  physical  location  and  alarm  information. 
This  may  have  been  due,  at  least  in  part,  to  the  limited  database  provided  for  Build  1 .5. 

f.  Data  fields,  in  particular  the  use  of  time  references,  were  inconsistent.  In  Maximo,  both 
Zulu  and  local  times  were  used,  increasing  the  likelihood  of  confusion  and  forcing  users 
to  mentally  translate  times.  FAA  practice  is  to  use  Zulu  time  as  a  standard  because 
operations  occur  across  a  number  of  time  zones.  This  will  be  even  more  significant  with 
the  centralization  of  control  at  the  three  OCCs. 

g.  Identifiers  were  used  inappropriately.  In  Maximo,  a  number  of  data  items  such  as 
personnel  lists  were  sorted  by  labor  category  code  or  other  identifiers  that  are  not  used  by 
specialists  and  may  not  even  be  known  by  them. 

h.  Users  could  not  transfer  data  between  applications.  With  three  major  applications  from 
three  different  sources,  it  is  essential  to  ensure  the  availability  of  data  between 
applications  to  minimize  the  need  for  data  reentry.  Build  1.5  did  not  adequately  support 
the  transfer  of  data  among  the  three  applications.  • 

The  users  indicated  that  much  of  the  needed  information  was  unavailable,  required  excessive 
effort  to  obtain,  or  was  inappropriately  formatted.  This  made  the  system  difficult  to  use  and 
interfered  with  user  task  performance.  Nearly  twice  as  many  comments  were  provided  on  this 
than  on  any  other  subcategory. 

3.3  Data  Entry 

Data  entry  is  the  process  of  inputting  information  into  a  system  through  the  use  of  text  fields  or 
graphical  controls.  Section  8.1  of  the  HFDG  (Wagner  et  al.,  1996)  provides  recommendations 
for  the  design  and  operation  of  data  entry  methods.  In  addition  to  the  concepts  addressed  in 
Section  3.2  of  this  document,  data  entry  encompasses  managing  and  preventing  errors  and 
minimizing  data  entry  activities. 

3.3.1  Text  Fields 


The  users  indicated  that  the  mechanism  to  modify  Maximo  logs  was  cumbersome.  Once  entered, 
data  were  not  easily  edited.  For  example,  to  edit  a  work  plan,  they  had  to  exit  the  log,  re-open  it, 
and  then  select  ADD.  Other  examples  included  editing  the  date  in  the  SER  form  and  editing 
shifts  in  the  Labor  module  (step  6. 1  g).  Another  concern  was  the  lack  of  a  spell  check  capability. 
OCC  operators  will  enter  significant  amounts  of  information  into  ETs,  and  the  ability  to  correct  a 
spelling  error  before  committing  it  to  a  log  is  highly  desirable. 
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3.3.2  Graphical  Controls 


Graphical  controls  permit  users  to  exercise  control  over  system  functions,  components,  or  data 
structures.  They  include  icons,  push  buttons,  radio  buttons,  check  boxes,  and  sliding  scales. 
Overall,  the  graphical  controls  in  Build  1 .5  (i.e.,  icons,  pushbuttons,  and  check  boxes)  were  used 
effectively.  Though  the  meaning  of  each  icon  was  not  immediately  apparent  to  all  users  (e.g.,  the 
Maximo  eraser  icon),  the  availability  of  pop-up  titles  when  users  placed  the  cursor  over  the  icon 
minimized  concerns.  The  system  should  be  reviewed  to  verify  that  pop-up  titles  or  conveniently 
accessed  descriptions  are  available  whenever  feasible. 

3.3.3  Error  Management  and  Prevention 

Error  management  and  prevention  is  a  significant  aspect  of  system  performance.  Shneiderman 
(1992)  reported  that  3 1%  of  professional  users  of  text  editing  software  made  errors  or  used 
inefficient  strategies. 

Most  users  in  this  evaluation  indicated  that  the  data  entry  methods  helped  to  minimize  errors. 
Users  easily  differentiated  between  text  entry  and  drop  down  lists.  However,  though  sample 
inputs  were  provided  near  many  data  input  fields,  they  were  not  available  consistently  across  all 
applications  or  screens.  For  example,  when  preparing  an  SER  (Step  4.6f),  there  was  no  format 
provided  for  entering  the  time  of  delay,  causing  one  user  to  ask  if  the  input  should  be  reported  in 
minutes,  hours,  or  some  other  format. 

The  human  factors  engineers  recommend  that  the  system  be  reviewed  to  ensure  that  guidance  is 
provided  near  the  input  field  whenever  feasible.  When  this  cannot  be  accomplished,  they  suggest 
the  provision  of  convenient  access  to  information  detailing  the  expected  data  input  format.  The 
HFDG,  Section  8.1.5,  provides  recommendations  regarding  error  management  (Wagner  et  al., 
1996). 

The  users  had  less  favorable  comments  on  detecting  and  correcting  errors.  Some  reported  that  it 
was  easy  to  detect  data  entry  errors,  but  only  one  found  it  easy  to  correct  these  errors.  One  user 
indicated  that  when  preparing  a  report  using  Maximo,  the  application  accepted  incorrectly 
formatted  data,  provided  no  feedback  that  the  data  were  formatted  inappropriately,  and  then 
failed  to  produce  a  report  without  providing  an  explanation.  All  of  the  participants  felt  that 
referencing  systems  specialists  by  labor  codes  instead  of  names  was  particularly  awkward.  One 
respondent  pointed  to  this  as  a  contributing  cause  of  data  input  errors.  In  addition,  users 
identified  the  preparation  of  SERs  as  a  complex  and  error  prone  process. 

All  participants  reported  that  NIMS  Build  1 .5  did  not  provide  a  smooth  return  to  operations  when 
recovering  from  system  failures  or  erroneous  actions.  For  example,  one  individual  reported 
accidentally  erasing  the  CET  table  by  pressing  the  Escape  key.  Another  user  recommended 
providing  the  ability  to  stop  processing  when  refreshing  ArcView  maps  because  they  require 
significant  processing  time.  This  would  enable  the  users  to  refresh  the  process  if  they  had  made  a 
wrong  selection.  Many  errors  required  the  application  to  be  reloaded,  and,  in  some  instances,  it 
was  necessary  to  reboot  the  workstation. 
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3.3.4  Minimum  User  Actions 


This  category  addresses  how  the  user  takes  actions  and  implements  decisions.  Users  should  not 
be  forced  to  complete  a  number  of  actions  to  achieve  a  simple  task.  Minimum  User  Actions  has 
two  components  -  minimizing  the  number  of  actions  required  and  minimizing  the  complexity  of 
the  action.  These  become  more  important  as  the  system  workload  and  stress  levels  increase. 
Complex  behaviors  tend  to  break  down  under  stress  before  simple  ones.  Emphasizing  minimum 
user  actions  can  significantly  reduce  the  risk  of  operational  errors  in  a  heavy-workload  OCC 
environment. 

This  category  generated  36  comments  from  users  and  observers.  Most  of  these  addressed  the 
excessive  number  of  actions  required  to  accomplish  a  task.  The  phrase  “too  many  steps”  was 
used  repeatedly.  There  were  two  main  reasons  reported  for  the  excessive  number  of  steps: 

a.  The  application  failed  to  auto-populate  fields  with  data  available  in  the  system.  This 
required  the  user  to  either  search  for  and  manually  transfer  data  or  reenter  the  data  before 
the  task  could  be  completed.  This  slowed  task  performance  and  increased  the  opportunity 
for  errors.  Failure  to  transfer  data  from  the  CET  to  the  ET  was  cited  as  a  significant 
example. 

b.  Assigning  a  field  specialist  to  an  ET  required  several  steps  to  locate  personnel  and  then  to 
make  the  assignment.  Much  of  the  data  and  parameters  (e.g.,  whether  the  normally 
assigned  specialist  is  already  assigned  or  whether  call  back  is  required)  were  already 
available  to  the  system. 

Action  complexity  refers  to  the  difficulty  resulting  from  how  an  action  is  performed.  For 
example,  a  key  press  is  less  complex  than  entering  text  or  locating  an  object  on  a  graphic  display, 
centering  a  trackball  tag  over  it,  and  then  clicking.  In  this  evaluation,  the  primary  issue  raised 
was  the  need  to  manually  enter  text  rather  than  selecting  it  from  a  short  list  or  having  other 
alternatives  available.  Another  case  of  unwanted  complexity  was  the  need  to  enter  the  word 
DATE  in  parentheses  before  actually  entering  the  date.  Because  this  is  a  date  field,  the  system 
should  recognize  the  date  without  the  preliminary  entry. 

The  relatively  large  number  of  comments  indicated  that  users  perceived  the  required  actions  as 
excessive  and  interfering  with  accomplishing  the  tasks.  The  issue  is  particularly  important  as 
workload  and  system  stress  increase.  When  performance  is  most  critical,  failure  to  minimize 
actions  required  can  result  in  serious  performance  degradation.  The  human  factors  engineers 
recommend  that  system  software  logic  be  considered  for  most  of  the  searching  and  sorting  tasks. 

3.4  Data  Display 

Data  display  refers  to  the  information  presented  on  a  computer  terminal  and  may  include  text, 
graphics,  or  a  combination  of  both.  It  is  closely  interrelated  to  the  components  addressed  under 
Basic  Screen  Design  and  Operation  (Section  3.2  of  this  document).  The  goal  of  any  display  is  to 
communicate  required  information  clearly  and  unambiguously  to  the  user.  Section  8.5  of  the 
HFDG  provides  specific  recommendations  regarding  the  design  of  both  text  and  graphics 
displays  (Wagner  et  al.,  1996). 
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3.4.1  Text 


This  area  is  restricted  to  the  legibility  of  text  in  displays  and  the  use  of  unique  and  descriptive 
labels.  Consistency  of  wording  and  labeling,  which  are  also  aspects  of  text  display,  were 
addressed  in  Section  3.2.1  of  this  document.  Users  reported  that  text  was  clear  and  easy  to  read 
across  all  applications.  They  did  not  consider  labeling,  itself,  a  major  issue.  However,  there 
were  instances  when  unique  labels  were  not  provided  and  when  labels  reflected  industry  and  not 
FAA  terminology.  The  human  factors  engineers  recommend  that  all  labels  be  reviewed  to  verify 
they  are  unique,  descriptive,  and  consistent  with  FAA  usage. 

3.4.2  Graphics 

Graphics  include  pictures,  geographic  displays,  diagrams,  graphs,  and  charts.  The  intent  of 
graphics  is  to  “facilitate  the  detection  of  relationships  among  variables,  comparisons  among  data 
sets,  and  the  detection  of  trends  in  the  data”  according  to  the  HFDG,  Section  8.5.5  (Wagner  et  al., 
1996,  p  8-107).  The  most  extensive  use  of  graphics  in  the  NIMS  Build  1 .5  interface  was  the  use 
of  maps  in  ArcView.  Most  of  the  users  indicated  that  these  maps  provided  accurate  service 
status  information.  However,  the  following  issues  were  raised. 

a.  The  sites  appeared  as  dots  without  labels,  and  the  users  had  difficulty  differentiating 
them.  This  was  more  significant  because  users  could  not  click  on  a  site  to  access  the  ET. 
Instead,  they  had  to  identify  the  site  in  ArcView,  go  to  MASS  to  determine  the  affected 
equipment,  and  then  locate  the  associated  CET  from  a  long  scrolling  list. 

b.  The  maps  did  not  provide  sufficient  detail  (e.g.,  streets). 

c.  Range  circles  could  be  added  on  top  of  each  other  without  getting  rid  of  the  previous 
instance.  This  was  confusing  to  the  users,  and  they  had  difficulty  clearing  them. 

d.  Regional  boundaries  were  not  clearly  visible. 

The  human  factors  engineers  recommend  that  future  systems  provide  easy  access  to  site 
identification  labels  and  associated  ETs,  additional  map  detail,  and  OCC  boundaries.  Guidelines 
for  common  map  symbology  are  provided  in  Symbol  Standardization  in  Airway  Facilities 
(Ahlstrom,  Cranston,  Mogford,  Ramakrishnan,  &  Birt,  1998). 

3.5  User  Access 


The  participants  noted  that  some  Maximo  screens  that  they  reviewed  were  not  required  for  them 
to  perform  their  current  duties.  For  example,  only  Program  Support  Specialists  and  Systems 
Maintenance  Office  Managers  require  access  to  labor  certifications  (step  3.1p).  The  users  were 
granted  access  to  the  entire  system  to  enable  them  to  evaluate  consistency  of  screen  design  and 
data  input  methods  across  all  modules  and  to  view  the  types  of  information  in  the  database. 

When  fielded,  it  is  anticipated  that  users  would  be  authorized  to  access  only  those  areas  of  the 
system  that  directly  supported  their  duties. 

One  area  of  the  system  that  raised  significant  concern  was  the  ability  to  modify  the  status  of  a  site 
in  ArcView.  For  example,  if  a  site  was  in  alarm,  any  user  could  select  that  site  and  change  the 
status  to  normal.  Confirmation  or  verification  of  identity  was  not  required.  If  the  capability  to 
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change  site  status  is  present  in  the  fielded  version  of  the  product,  the  human  factors  engineers 
recommend  that  the  list  of  personnel  authorized  to  make  such  a  change  be  carefully  reviewed. 

3.6  Help 

A  good  on-line  help  utility  can  significantly  improve  the  usability  of  a  system  and  can  aid  users 
in  error  recovery.  However,  the  NIMS  Build  1.5  test  suite  provided  little  on-line  help  to  users. 
MASS  and  Maximo  did  not  incorporate  a  help  utility,  and  the  information  provided  by  ArcView 
was  not  FAA  specific  and,  therefore,  of  limited  value. 

Human  factors  guidelines  suggest  that  on-line  help  should  always  be  accessible  to  the  user, 
provide  a  context-sensitive  search  capability,  and  be  sensitive  to  the  context  within  which  it  was 
requested  (Wagner  et  al.,  1996).  Weinschenk  and  Yeo  (1995)  indicate  that  online  help  should 
include  information  that  is  urgent,  specific,  changed  frequently,  is  complexly  organized,  or 
incorporates  large  amounts  of  reference  information. 

4.  Conclusions 


The  purpose  of  this  study  was  to  evaluate  the  NIMS  Build  1 .5  Specialist  interface.  The  human 
factors  engineers  aggregated  the  participant  comments  into  human  factors  categories  and 
identified  those  issues  most  likely  to  affect  the  ability  of  users  to  effectively  perform  their  tasks. 

This  evaluation  was  subject  to  a  number  of  known  limitations,  which  could  significantly  affect 
performance  and  its  evaluation.  The  software  was  still  in  development,  so  the  three  applications 
were  not  fully  integrated  and  not  all  functionality  was  provided.  Some  of  the  users  anticipated  a 
more  complete  product,  and  this  may  have  affected  their  evaluation.  Only  one  user  and  one 
observer  had  previous  training  or  experience  with  Maximo.  Also,  not  all  users  had  experience 
with  the  MASS  software  and  none  had  been  exposed  to  ArcView.  Whereas  the  software  and 
system  were  known  to  be  incomplete,  the  human  factors  engineers  considered  the  issues  noted  in 
this  report  to  be  relevant  to  any  future  development  of  a  centralized  AF  maintenance  monitoring 
and  control  system. 

4.1  Major  Issues 

Based  on  the  number  of  comments  made  in  each  of  these  categories,  the  human  factors  engineers 
identified  four  significant  issues  that  warrant  investigation.  These  areas  include  application 
integration,  information  availability,  error  messages  and  management,  and  feedback. 

4.1.1  Application  Integration 

With  three  major,  disparate  applications  involved,  a  significant  issue  is  integrating  these  into  a 
single  functional  system.  Because  two  of  these  are  COTS  items  from  different  vendors  and  one 
is  government  developed,  there  is  little  commonality  in  their  underlying  structures  or  approach  to 
presentation  of  data  and  operational  control.  User  and  observer  comments  in  several  sections 
clearly  indicated  that  there  was  insufficient  transfer  of  data  across  the  interfaces. 
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4. 1 .2  Information  Availability 


For  the  users  in  this  evaluation,  Maximo  was  the  most  extensively  employed  application  and 
most  comments  addressed  this  software.  In  general,  the  users  did  not  consider  the  data 
availability  appropriate  to  their  needs.  As  tested,  they  thought  that  the  system  contained 
extraneous  and  rarely  used  items  on  many  screens.  The  result  was  that  task  completion  was 
difficult,  slower,  and  potentially  error  prone.  Given  the  likelihood  of  high  workload  demands  in 
a  centralized  monitoring  and  control  facility,  significant  effort  is  warranted  to  define  and  design 
screen  data  content.  This  redesign  should  be  based  on  detailed  task  analyses  and  extensive 
contact  with  AF  personnel. 

The  users  felt  that  the  Build  1 .5  version  of  Maximo  was  inadequately  adapted  to  the  FAA 
environment.  Data  presentation,  data  access,  and  terminology  are  examples  of  areas  that  need 
improvement. 

4.1.3  Error  Messages  and  Management 

The  users  found  error  messages  largely  undecipherable.  Most  of  the  error  messages  seemed  to  be 
directed  at  application  programmers  rather  than  at  AF  users.  They  gave  the  user  little  assistance 
in  correcting  the  error.  The  users  found  that  the  software  also  appeared  to  have  problems  with 
certain  kinds  of  error  detection.  A  number  of  fields  were  found  that  would  accept  incorrect 
entries  without  warning  the  user.  In  general,  error  messages,  detection,  and  management  need 
significant  attention  before  this  program  is  fielded. 

4.1.4  Feedback 


The  users  indicated  that  the  lack  of  feedback  to  their  inputs  and  actions  was  a  major  issue.  For 
many  user  actions,  there  was  no  way  to  determine  if  the  action  had  been  accepted  or  if  there  was 
a  failure  due  to  an  input  error  or  a  system  problem.  Immediate  feedback  as  to  the  success  of  an 
action  or  the  state  of  the  system  in  performing  some  extended  function  is  critical  to  successful 
system  performance. 

4.2  Overall  Conclusions 


The  issues  identified  in  the  current  evaluation  are  typical  of  those  that  should  be  addressed  in  any 
system  intended  to  function  in  a  remote  monitoring  and  control  facility.  At  its  tested  maturity 
level,  Build  1.5  was  not  capable  of  supporting  an  OCC  Specialist  in  an  operational  environment. 
This  evaluation  identified  many  areas  that  need  improvement.  In  spite  of  these  limitations,  some 
of  the  users  felt  that  Build  1 .5  could  be  improved  to  meet  their  needs.  The  major  issues 
identified  were  application  integration,  information  availability,  error  messages  and  management, 
and  feedback.  The  human  factors  engineers  concluded  that  the  majority  of  issues  identified  in 
this  evaluation  could  be  mitigated  through  the  application  of  sound  human  engineering 
principles. 

It  should  be  noted  that  these  recommendations  were  provided  to  the  program  office  in  a  timely 
manner  after  the  conclusion  of  this  study.  Final  formal  publication  of  this  report  was  delayed  due 
to  competing  priorities  for  resources. 
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ACRONYMS 


AF 

Airway  Facilities 

CET 

Candidate  Event  Ticket 

CHI 

Computer  Human  Interface 

COTS 

Commercial-Off-The-Shelf 

ET 

Event  Ticket 

FAA 

Federal  Aviation  Administration 

HFDG 

Human  Factors  Design  Guide 

ID 

Identification 

MASS 

Maintenance  Automated  System  Software 

NAS 

National  Airspace  System 

NIMS 

National  Airspace  System  Infrastructure  Management  System 

NPF 

NIMS  Premier  Facility 

OCC 

Operations  Control  Center 

SER 

Significant  Event  Report 

SQL 

Sequel 
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Appendix  A 

CHI  Evaluation  Test  Scripts 

1.0  MASS  familiarization 
Description: 

This  test  will  provide  familiarization  with  the  start  up  and  operation  of  MASS. 

Objective: 

To  become  familiar  with  the  function  and  operation  of  MASS. 

Limitations: 

The  events  will  be  induced  using  a  non-operational  database. 

Input: 

COTS  software,  FAA  provided  software,  configuration  information. 

Output: 

Alarm  indication  in  the  MASS  table. 

Success  Criteria: 

User  should  be  able  to  acknowledge,  mask,  and  filter  alarms. 

Procedures: 

Test  Steps:  Expected  Results:  Comments: 

1.0  System  Logon 

a)  Press  Ctrl+Alt+Delete  and  then  Logon  dialog  box 
“OK”. 

b)  Depending  on  position,  use  one  **Caution  logon  is  case  sensitive  ** 
of  the  following  logons  for  both  the 

USERID  and  PASSWORD: 
occuserl,  occuser2,  occuser3, 
noccuser,  socuser,  or  wcuser. 

1.1.  MASS  client  log  on 

a)  Launch  MASS  by  clicking  on  MASS  logon  window  is  displayed, 
the  MASS  icon. 
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Test  Steps: 

Expected  Results: 

b)  Enter  USERID  (ea27coc)  and 
PASSWORD  (PASSWORD)  in 
logon  window,  click  on  OK  button. 

**Caution  logon  is  case  sensitive  ** 

System  Monitor  window  displayed. 
Alarms  list  is  displayed  in  the 
event  log. 

c)  Maximize  both  windows. 

Information  about  alarm  event 
include: 

1 .  Cnt:  (Alarm  Count) 

2.  PR:  (Priority) 

3.  Status:  * 

4.  Site:  (Facility  Ident) 

5.  Type:  (Facility  Type) 

6.  LUID:  (Logical  Unit  &  IDU) 

Cr:  (Criticality) 

Roll-up  (+)  ~  more  than  one  event . 
Parameter  Description:  Event 

level  (Low,  Actual,  high) 

d)  Select  an  unacknowledged 
alarm/alert  from  the  list  and 
click  on  the  Acknowledge 

Alarm  Icon  (checkmark 
symbol)  in  the  toolbar. 

Alarms/ Alert  dialog  box  is 
displayed 

e)  Select  alarm/alert  from  Alarm 
&  Alert  dialog  box  and  click 
“OK”. 

Observe  the  alarm  is  acknowledged 
and  the  list  is  updated. 

f)  Click  on  the  “Mask”  Icon  in 
toolbar. 

The  Mask  dialog  box  will  be 
displayed.  This  function  is  used  to 
enable  alarm/alert  masks  for  a 
specific  time  frame  on  the  facilities 
and  parameters  selected 

g)  Click  “CANCEL” 

The  mask  dialog  box  closes. 

h)  Click  on  the  “Filter”  Icon  in 
toolbar. 

The  filter  dialog  box  is  displayed. 

i)  Double  click  on  the 
Acknowledged  Alarms  and 

Alerts  in  the  Condition  Box 

Acknowledged  alarms  and  alerts  are 
displayed  in  the  Filter  Box. 

j)  Click  “OK” 

Acknowledged  alarms/alerts  are 
filtered  and  not  displayed  in  list. 

k)  Click  on  the  Alarms  list.  Site 

List,  and  Unack  list. 

Each  list  type  is  displayed. 

Comments: 


*  A  =  Alarm 
a  =  Alert 
N  =  Normal 
S  =  Status  message 
Check  =  Acknowledged 
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2.0  ArcView  familiarization 


Description: 

ArcView  provides  a  graphic  representation  of  alarm/alert  status  of  the  NAS.  This  procedure  will 
provide  familiarization  with  the  start  up  and  operation  of  ArcView. 

Objective: 

To  become  familiar  with  the  function  and  operation  of  ArcView. 

Limitations: 

Only  non-intrusive  events  will  be  induced  using  non-operational  systems. 

Input: 

COTS  software,  FAA  provided  software,  configuration  information. 

Output: 

Alarm  indications  on  graphic  display,  facility  information,  and  display  feature  customization. 

Success  Criteria: 

User  should  be  able  to  configure  a  display  to  view  desired  alarms  and  retrieve  facility 
information 

Procedures: 


Test  Steps: 

Expected  Results: 

Comments: 

2.0  System  Logon 

a)  Press  Ctrl+Alt+Delete  and  then 
“OK”. 

Logon  dialog  box 

b)  Depending  on  position,  use  one 
of  the  following  logons  for  both 
the  USERID  and  PASSWORD: 
occuserl,  occuser2,  occuser3, 
noccuser,  socuser,  or  wcuser. 

**Caution  logon  is  case  sensitive  ** 

2.1.  Arcview  client  log  on 

a)  Double  click  Internet  Explorer 
Icon. 

NPF  window  is  displayed. 

b)  Click  on  “National  Facilities 
Monitoring  Display” 
hyperlink. 

ArcView  logon  box  is  displayed. 

Test  Steps: 


Expected  Results: 


Comments: 


c)  Enter  the  USERID  and  Allow  approx.  15  minutes  for 

PASSWORD  entered  in  step  application  to  load. 

2.0  b  press  “OK” 

d)  Select  and  then  Deselect  each  Observe  the  changes  in  the  display 
item  in  the  Main  View  toolbox  presentation. 

on  the  left  side  of  the  screen. 

e)  Select  the  Add  Group  Icon  from  Drop  down  list  is  displayed 
the  top  left  hand  comer  of  the 

tool  menu. 

f)  From  drop  down  lists  Select:  Allow  approx.  1  minute  for  the  Radar 

Terminal  -  “OK”  group  to  be  added  to  the  system. 

Radar- “OK” 

Remote -“OK” _ 

g)  Select  ASR  check  box  in  menu 
area  on  left  side. 

h)  Select  the  Zoom  In  icon  from  Magnified  view  of  the  area  selected 
the  tool  bar  (+).  Click  and  Drag  should  be  displayed. 

the  tool  to  around  the 
Washington  DC  area,  including 
Northern  VA  (Dulles). 

i)  Adjust  the  ICON  size  using  the  Facility  Icons  change  size. 

“Increase  Icon”  and  “Decrease 
Icon”  Icon  in  the  toolbar. 

j)  Select  the  ASR  check  box  area.  A  raised  box  should  appear  around 

the  ASR  checkbox 

k)  Click  on  the  “i”  (information)  Facility  information  dialog  box  is 
Icon  in  toolbar  and  place  it  over  displayed. 

the  center  of  a  facility  icon  on 
the  graphic  display.  Click 

l)  Click  on  the  “Adjacent  Radar”  Adjacent  radar  coverage  area 

Icon  toolbar  and  place  it  over  displayed, 
the  center  of  a  facility  icon  on 
the  graphic  display.  Click 

m)  Select  “Display  Areas”  from  Graphic  display  reverts  back  to  the 
menu  bar  and  CONUS  from  CONUS  view. 

drop  down  list. 
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3.0  Maximo  familiarization 


Description: 


This  test  will  provide  familiarization  with  the  start  up  and  operation  of  Maximo 


Objective: 

To  become  familiar  with  the  function  and  operation  of  Maximo. 


Limitations: 

Only  non-intrusive  events  will  be  induced  using  non-operational  systems. 

Input: 

COTS  software,  FAA  provided  software,  configuration  information. 


Success  Criteria: 

Become  familiar  with  Maximo  functional  screens. 


Procedures: 


Test  Steps: 

Expected  Results: 

Comments: 

3.0  System  Logon 

a)  Press  Ctrl+Alt+Delete  and  then 
“OK”. 

Logon  dialog  box 

b)  Depending  on  position,  use  one 
of  the  following  logons  for  both 
the  USERID  and  PASSWORD: 
occuserl,  occuser2,  occuser3, 
noccuser,  socuser,  or  wcuser. 

**Caution  logon  is  case  sensitive  ** 

3.1.  Maximo  client  log  on 

a)  Double  click  Internet  Explorer 
Icon. 

NPF  window  is  displayed. 

b)  Click  on  “Infrastructure 
Resource  Management 

System”  hyperlink. 

Maximo  logon  box  is  displayed. 

c)  Enter  the  USERID  and 

PASSWORD  entered  in  step 

3.0  b,  press  “OK” 

d)  Select  “Event  Ticket”  Icon  and 
select  “Event  Tickets 
Tracking”  from  the  drop  down 
list. 

Event  Ticket  screen  is  displayed. 

Test  Steps: 


Expected  Results: 


Comments: 


e)  Select  each  tab  in  the  event  Each  tab  will  display  the  fields 

ticket  area  and  explore  the  associated  with  that  tab. 

fields. 

f)  Select  the  “Event  Ticket”  Icon  Candidate  Event  Ticket  screen  is 

from  the  menu  on  the  left  side  displayed, 
of  the  screen.  Select 
“Candidate  Event  Tickets” 
from  the  drop  down  list. 

g)  Select  each  tab  in  the  Candidate  Each  tab  will  display  the  fields 
Event  Ticket  area  and  explore  associated  with  that  tab. 

the  fields. 

h)  Select  the  “Preventive  Preventive  Maintenance  screen  is 

Maintenance”  Icon  from  the  displayed. 

menu  on  the  left  side  of  the 
screen.  Select  “Preventive 
Maintenance”  from  the  drop 
down  list. 

i)  Select  each  tab  in  the  Preventive  Each  tab  will  display  the  fields 

Maintenance  area  and  explore  associated  with  that  tab. 
the  fields. 

j)  Select  the  “Equipment”  Icon  Equipment  screen  is  displayed. 

from  the  menu  on  the  left  side 
of  the  screen.  Select 
“Equipment”  from  the  drop 
down  list. 

k)  Select  each  tab  in  the  Each  tab  will  display  the  fields 

Equipment  area  and  explore  the  associated  with  that  tab. 
fields. 

l)  Select  the  “Equipment”  Icon  Operating  Location  screen  is 

from  the  menu  on  the  left  side  displayed, 
of  the  screen.  Select 
“Operating  Location”  from  the 
drop  down  list. 

m)  Select  each  tab  in  the  Each  tab  will  display  the  fields 

Operating  Location  area  and  associated  with  that  tab. 
explore  the  fields. 

n)  Select  the  “Plans”  Icon  from  the  Plans  screen  is  displayed, 
menu  on  the  left  side  of  the 

screen.  Select  “Job  Plans” 
from  the  drop  down  list. 
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Test  Steps: 


Expected  Results: 


Comments: 


o)  Select  each  tab  in  the  Job  Plans  Each  tab  will  display  the  fields 

area  and  explore  the  fields.  associated  with  that  tab. 

p)  Select  the  “Labor”  Icon  from  Labor  screen  is  displayed, 
the  menu  on  the  left  side  of  the 

screen.  Select  “Labor”  from 
the  drop  down  list. 

q)  Select  each  tab  in  the  Labor  Each  tab  will  display  the  fields 

area  and  explore  the  fields.  associated  with  that  tab. 

r)  Select  the  “Labor”  Icon  from  the  Calendar  screen  is  displayed. 

menu  on  the  left  side  of  the 
screen.  Select  “Calendar” 
from  the  drop  down  list. 

s)  Select  each  tab  and  explore. 
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4.0  Alarm  Detection,  Candidate  Event  Ticket  Generation,  and  Event  Ticket  Generation. 

Description: 

>  Simulate  ASR-9  alarm  from  MASS 

>  View  the  alarm  in  ArcView 

>  Auto  creation  of  Candidate  Event  Ticket  (CET) 

>  Generation  and  population  of  an  Event  Ticket  (ET) 

>  Simulated  repair  and  event  closure 

Objective: 

The  objective  of  this  test  is  to  verify  that  a  CET  is  automatically  generated  from  a  MASS  alarm. 
An  ET  can  be  generated  from  a  CET. 

Limitations: 

Only  non-intrusive  events  will  be  induced  using  non-operational  systems. 

Input: 

COTS  software,  FAA  provided  software,  configuration  information. 

Output: 

Facility  alarm  messages  are  populated  in  a  Candidate  Event  Ticket. 

Success  Criteria: 

Automatically  populate  alarm  information  in  the  Candidate  Event  Ticket  fields. 

Procedures: 


Test  Steps: 

Expected  Results: 

Comments: 

4.0  System  Logon 

a)  MASS,  ArcView,  and  Maximo 
should  be  up  and  running 
before  proceeding.  Follow  the 
System  Logon  procedures  in 
scenario  1.0,  2.0,  and  3.0. 

4.1  MASS  Alarm 

a)  Advise  one  of  the  NPF  staff  that 
a  simulated  ASR  alarm  is 
required. 

b)  Observe  the  MASS  alarm  list 
and  verify  the  alarm  is 
generated. 
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Test  Steps: 


Expected  Results: 


Comments: 


c)  Acknowledge  the  alarm  in 
MASS  by:  selecting  the  alarm 

Clicking  the  Acknowledge 
Icon,  selecting  the  alarm  in 
dialog  box  list,  and  “OK” 


d)  Observe  the  alarm  in  ArcView: 
ensure  the  Terminal-Radar- 
Remote  group  is  displayed.  If 
not  follow  procedure  2. 1  .e 


e)  Zoom  in  on  the  alarm  using  the 
Zoom  in  tool  (+). 


f)  Select  the  ASR  check  box  area. 


g)  Select  the  information  Icon  (i) 
and  click  on  the  facility  alarm 


4.2  CET  Generation 


a)  In  Maximo,  Select  the  Event 
Ticket  Icon  and  Candidate 
Event  Ticket  from  the  drop 
down  list. 


b)  Verify  that  the  ASR  CET  was 
generated  for  the  alarm  that  was 
acknowledged  in  MASS. 

c)  Select  the  ASR  CET  and  select 
the  “CET  Detail”  tab. 


4.3.  Overview  the  history  CETs 
and  running  CET  history 
report 


a)  Click  on  CET  Table  tab 


b)  To  generate  a  history  CET 
report,  Select  File  -Run 
Reports. 


c)  Select  the  “CETHIST”  report 
and  “Run”. 


d)  Select  Preview  option,  “OK”. 


The  alarm  list  should  update 
indicating  the  change  in  status. 


A  raised  box  should  appear  around 
the  ASR  checkbox 


The  facility  details  should  be 
displayed. 


The  CET  Table  is  displayed 


Review  the  Detail  information. 


CET  Report  window  is  displayed. 


SQR  Option  window  is  displayed. 


“Parameters  -  CETHIST”  window 
is  displayed 


Test  Steps: 


Expected  Results: 


Comments: 


e)  Enter  the  site  location  (IAD)  and  CET  History  Report  is  displayed, 
the  site  type  (ASR),  at  Start 
tab  enter  date,  and  at  End  tab 
enter  date.  Click  “OK” 


f)  Review  the  report  and  Close  it 
by  clicking  the  lower  “X”  in  the 
top  right  hand  comer. 


g)  Select  the  “CET  Table”  tab 
and  select  the  ASR  CET. 


h)  Click  on  the  “Generate  Event  Make  note  of  the  ticket  # 
Ticket”  button.  Select  “OK” 
and  “YES”  to  go  to  the  ET. 


4.4  Event  ticket  population. 


a)  In  the  Event  Ticket  Details  Box 
just  to  the  Right  of  the  ticket 
number  enter  “ YOUR  NAME 
ASR-9  Repair” 


b)  In  the  Responsibility  area  click 
the  select  Ticket  Priority 
Detail  button  (***)  and  “3” 
from  the  drop  down  list.  “OK” 


c)  Select  the  OCC  Specialist 
detail  button  (***)  to  display 
the  list  of  OCC  Specialist  and 
select  one  to  the  ticket.  “OK” 


d)  In  job  details  area  select  event 
type  and  “EM”  from  the  drop 
down  list.  “OK” 


e)  Select  outage  class  and  “FL” 
from  the  drop  down  list.  “OK” 


f)  In  Schedule  Information  area 
select  actual  outage  details 
button.  “OK” 


g)  In  Responsibility  area  select 
Assign  a  Specialist.  And 
choose  one  from  the  list. 
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Test  Steps: 


Expected  Results: 


Comments: 


h)  Select  Contact  Specialist  and  Assume  the  page  was  received  and 
push  Page  Method.  “OK”.  the  specialist  is  responding. 

Close  all  pager  associated 

dialog  boxes  and  assume  the 
specialist  was  successfully 
paged. 

i)  Re-Select  the  specialist  and  Make  note  of  specialist’s  labor 

press  Make  Assignment.  code. 

“CLOSE” 

j)  Select  Action-Change  Status 

from  the  menu  bar  and  select 
Approved. 

k)  Select  Electronic  Log  Notes 
icon  from  the  top  tool  bar  and 
enter  “Specialist  contacted 
and  responding”.  “OK” 

4.5  Specialist  ET  population.  Assume  you  are  the  specialist  in 

the  field. 

a)  Select  View-Saved  Queries  The  first  ticket  assigned  to  this 
from  the  menu  bar.  Select  the  specialist  is  displayed, 
specialist’s  labor  code  from 

step  J  and  hit  Execute. 

b)  Select  the  Global  List  icon  from 
the  toolbar.  All  the  open  ET’s 
assigned  to  this  specialist  are 
displayed.  Select  the  ticket  you 
generated.  “OK”. 

c)  Select  Action-Change  Status  Note  the  status  change 
from  the  menu  bar  and  select 

Inprogress. 

d)  Select  Electronic  Log  Notes 
icon  from  the  top  tool  bar  and 
enter  “system  repaired  and 
restored”.  “OK” 

e)  Select  Action-Change  Status  Note  the  status  change, 
from  the  menu  bar  and  select 

Completed. 

4.6  OCC  Review  and  SER 
generation. 

a)  Select  File-Run  Reports- 
Electronic  log  notes  from  the 
menu  bar.  “RUN” 
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Test  Steps: 


Expected  Results: 


Comments: 


b)  Enter  the  ticket  number  from  The  log  is  displayed  for  review, 
the  ET  generated  4.3 .c.  Also 

select  the  Application  tab  and 
enter  “WOTRACK”  -“OK” 

c)  Close  the  log  by  clicking  the 

lower  “X”  in  the  top  right  hand 
comer. 

d)  Click  in  the  SER  box  and  type 
“Y”  Save  file. 

e)  Select  Create  SER  pushbutton 
and  populate  SER  fields  (there 
are  3  tabs).  When  completed 
select  Publish  Version. 

f)  Select  File-Run  Reports-SER  The  SER  is  displayed. 

from  the  menu  bar.  Enter  the 
ET  #.  “RUN” 

g)  Close  the  SER  by  clicking  the 
lower  “X”  in  the  top  right  hand 
comer 

h)  Select  Action-Change  Status  Note  status  change, 
from  the  menu  bar  and  select 

Close. 
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5.0  Building  a  New  Event  Ticket 


Description: 

This  test  will  demonstrate  preparing  a  new  Event  Ticket. 

>  Access  Event  Ticket  Records 

>  Identity  Job  Operations 

>  Identify  Labor 

>  Identify  Needed  Materials 

>  Identify  Tools 

>  Assign  Event  Ticket 

>  Approve ,  Complete,  and  Close  an  Event  Ticket 
Objective: 

The  objective  of  this  test  is  to  demonstrate  NPF  capabilities  of  building  a  Event  Ticket. 

Requirements: 

SLS637,  SLS647,  SLS648,  SLS651 

Limitations: 

Not  known  at  this  time. 

Input: 

MAXIMO  software,  FAA  provided  software,  configuration  information. 

Output: 

Displays  tools  that  are  used  to  build  Event  tickets. 

Success  Criteria: 

The  NPF  should  provide  tools  to  build  Event  Ticket. 


Procedures: 


Test  Steps: 

Expected  Results: 

Comments: 

5.1.  Access  Event  Ticket  Records: 

a)  From  MAXIMO  main  menu,  Click  on 
“Event  Tickets”  module  icon.  Select 
“Event  Ticket  Tracking”. 

Event  Ticket  Tracking  window  is 
displayed. 

b)  Click  on  “Insert  Record  with 

AutoNumber”  button  (far  left  icon  on 
tool  bar) 

Event  Ticket  number  field, 
Reported  By  Date  field,  Event 
Ticket  Status  field,  and  Status 

Date  field  are  auto-populated. 
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Test  Steps: 


Expected  Results: 


Comments: 


c)  Click  on  Location  field  detail  button.  Drilldown  willdow  opens. 
Drilldown  to  the  following  location: 

AEA-REG,  CBS-SMO,  IAD-SSC, 

IAD-SSC,  and  IAD-ASR.  “OK”. 

d)  In  the  Event  Ticket  Details  Box  just  to 
the  Right  of  the  ticket  number  enter 

“ YOUR  NAME  ASR-9  PM” 

e)  In  the  Responsibility  area  click  the 

select  Ticket  Priority  Detail  button 
(***)  and  “2”  from  the  drop  down 
list.  “OK” 

f)  Select  the  OCC  Specialist  detail 

button  (***)  to  display  the  list  of 
OCC  Specialist  and  select  one  to  the 
ticket.  “OK” 

g)  In  job  details  area  select  event  type  and 
“PM”  from  the  drop  down  list.  “OK” 

h)  Select  outage  class  and  “FL”  from  the 
drop  down  list.  “OK” 

5.2.  Identify  Job  Operations:  (Plans) 

a)  Click  on  the  detail  button  in  Job  Plan  Plans  window  is  opened. 

field,  select  an  existing  job  plan. 

“OK”.  Job  Plan  field  is  auto-populated. 

b)  Select  Plans  tab.  Operations  steps,  Labor,  and 

Material  sections  are  auto- 
populated. 

5.3.  Assign  Event  Ticket: 

a)  In  Responsibility  area,  select  Assign  a  Record  the  specialist’s  labor 

Specialist  and  choose  one  from  the  code, 
list. 

b)  Select  the  Labor  icon  and  select 
Calendars  from  the  drop  down  list. 

c)  Type  the  Specialist’s  labor  code  into 
the  Calendars  field  and  “ENTER”. 

d)  Click  on  the  details  button  (***)  for 
today’s  date  to  get  the  Specialists  shift 
details.  “OK” 
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Test  Steps: 


Expected  Results: 


Comments: 


e)  Select  Event  Tickets  icon  and  Event 
Ticket  Tracking. 


f)  In  Responsibility  area  select  Assign  a 
Specialist.  And  choose  one  from  the 
list  and  select  Make  Assignment. 
“OK” 


5.4.  Event  Ticket  status  changes: 


a)  Click  on  Action-Change  Status. 
Select  Approved. 


b)  Enter  a  Scheduled  Start  time  by 

clicking  on  detail  button  and  “OK”. 


c)  Enter  an  Estimated  Duration  for  the 
event. 


d)  Click  on  Action-Change  Status. 
Select  Initiate. 


f)  Click  on  Actuals  tab. 


g)  Click  on  the  Labor  sub-tab  and  then 
click  OPS  field  on  the  first  line.  Click 
the  detail  button  and  select  an  item 
from  the  drop  down  list 


h)  Tab  to  next  field  and  select  Specialist. 


i)  Tab  to  name  field. 


j)  Tab  to  Start  Time  and  enter  start  time; 
tab  to  Finish  Time  and  enter  finish 


k)  Click  on  ET  tab  and  then  click  on 
Action  -  Change  Status.  Select 
Complete. 


1)  Click  on  Cost  tab  and  review 
information. 


1)  Click  on  Action  -  Change  Status. 
Select  Close. 


m)  Select  Action-View  status  history. 


Initiate  window  is  displayed. 


Actuals  window  is  displayed. 
Operations  fields  are  auto- 
populated  from  the  job  plan. 


The  Select  Value  window  is 
displayed  with  a  list  of  specialist. 


Labor  Code  field  is  auto- 
populated  with  “CE02RP” 


Field  auto-populates 


Hours  field  and  line  cost  field 
auto-populates. 


Complete  window  is  displayed. 


Close  window  is  displayed. 


The  Ticket  Status  history  is 
displayed. 


OCC  will  close  after  all 
coordination  is  completed. 
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6.0  Work  Force  Schedule  Creation  and  Tracking 
Description: 

The  NPF  should  provide  tools  to  create  and  track  work  force  schedule. 

Objective: 

The  objective  of  this  test  is  to  verify  that  NPF  demonstrates  capabilities  of  work  force  schedule 
creation  and  tracking. 

Requirements: 

SLS627 

Limitations: 

Only  non-intrusive  events  will  be  induced  using  non  operational  systems. 

Input: 

COTS  software,  FAA  provided  software,  configuration  information. 

Output: 

Displays  calendar  with  listings  of  work  periods,  date,  shifts,  and  holidays. 

Success  Criteria: 

The  NPF  provides  means  to  demonstrate  work  force  schedule  tracking. 

Procedures: 


Test  Steps: 

Expected  Results: 

Comments: 

6.1  Creating  a  shift: 

a)  Click  Labor  icon  on  Maximo  main 
menu. 

Labor  Application  menu  displays 
following  selection: 

Labor 

Labor  Reporting 

Calendars 

Work  Assignment  Queue 

b)  Select  Calendars 

Calendars  window  displays. 
Default  screen  is  the  Calendar 
tab  screen. 

c)  Select  Actions  -Define/ Apply  Shifts 

Define/ Apply  Shifts  window 
opens 

d)  Click  Insert  Row  button  and  enter  a 
shift  name  (i.e.,  your  initials). 

e)  Tab  to  next  field  and  enter  a  shift 
description  (i.e.,  test  schedule) 
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Test  Steps: 


f)  Tab  to  next  field  and  enter  a  Start  Day 
by  clicking  on  detail  button  and 
selecting  a  day  from  the  list,  “OK”. 


g)  Tab  to  next  field  and  enter  days  in  the 
shift  pattern  and  click  save. 


h)  Highlight  the  newly  entered  shift  row 
and  click  on  Define  Pattern. 


i)  Enter  Start  Time  and  End  Time  for 
each  day  of  the  pattern.  Tab  past  days 
off  (not  worked). 


j)  When  all  days  of  pattern  are 
completed,  click  on  OK  and  then 
Close. 


6.2  Creating  a  calendar: 


a)  Click  on  Calendar  tab  and  then  click 
on  Insert  icon  on  tool  bar  (top  left). 


b)  Enter  a  calendar  name  (i.e.,  your 
first  name)  in  the  Calendar  field  and 
press  tab. 


c)  Enter  a  short  description  of  the 
calendar  and  then  tab  to  next  field. 


d)  Enter  a  start  date  by  clicking  on  the 
detail  button  and  selecting  a  start  day 
and  then  OK. 


e)  Tab  to  the  End  Date  and  enter  a  End 
date  by  clicking  on  the  detail  button 
and  selecting  a  end  day,  click  OK,  and 
press  tab  key. 


f)  Click  Save  icon  on  tool  bar. 


g)  Select  Action  -  Define/ Apply  Shifts 


h)  Select  the  previously  created  shift  and 
click  on  Apply  Shifts  button 


i)  Click  OK  and  the  close  Define/ Apply 
Shifts  window  by  clicking  on  “X”  in 
upper  right  comer. 


j)  Choose  a  day  and  click  on  the  detail 
button. 


k)  Click  OK  to  return  to  Calendar. 


Expected  Results: 


Comments: 


Define/ Apply  Shifts  window 
opens. 


Apply  Shifts  window  opens 


Calendar  is  populated  with  The  number  displayed  by  each 
shift  data  day  of  the  month  is  the  number 


of  hours  worked. 


The  View/Modify  Work  Periods 
window  opens,  displaying  the 
work  schedule  for  that  day. 
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Test  Steps: 


Expected  Results: 


Comments: 


6.3  Modifying  Calendar  by  Adding 
Holidays: 


a)  Select  Action  -  Define/ Apply 
Holidays. 


b)  Select  the  appropriate  holiday  and 
click  Apply  Holidays. 


c)  Click  OK  and  then  cancel  to  close 
Define/Apply  Holidays  window. 


d)  Highlight  holiday  day  and  click  on 
detail  button.  “OK”. 


6.4  Modify  Calendar  by  Adding  Non¬ 
work  day. 


a)  Click  on  Actions  -  Apply  Non- 
Working  Time. 


b)  Click  on  detail  button  in  Non-Working 
Type  field  and  select  a  type  and  click 

OK. 


c)  Click  on  detail  button  in  Start  Date 
field  and  select  a  start  date  from  the 
Date/Time  calendar  window,  and  click 

OK. 


Define/ Apply  Holidays  window 
opens. 


Apply  Holidays  window  opens 


Observe  that  the  holiday  date 
displays  0  hours  worked. 

View/Modify  Work  Periods 
window  opens,  displaying 
“Holiday”. 


Apply  Non-Working  Time 
window  opens. 


d)  Click  on  detail  button  in  End  Date 
field  and  select  a  end  date  from  the 
Date/Time  calendar  window,  and  click 

OK. 

e)  Click  OK  in  the  Apply  Non-Working  Non- working  time  displayed  on 

Time  window  calendar  as  0  work  hours  for 

each  day  identified. 

f)  Click  Work  Periods  tab  to  display 
schedule  in  a  tabular  format. 
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Appendix  B 

Post-Usability  Evaluation  Questionnaire 


NIMS  Build  1.5  Usability  Questionnaire 


Purpose: 

The  following  questions  are  designed  to  assess  your  opinion  of  the  usability  of  NIMS  Build 

1 .5  in  support  of  your  AF  duties. 

Instructions: 

1 .  Read  each  item  carefully.  Please  restrict  your  answers  to  NIMS  version  1.5. 

2.  Check  the  box  that  most  accurately  reflects  your  experience  using  NIMS  Build  1 .5.  Each 
item  provides  a  statement  followed  by  2  choices— either  True,  or  False.  If  you  found  that 
a  feature  was  unacceptable  at  any  time  during  the  evaluation,  please  choose  False  as  your 
response  and  explain  the  circumstances. 

3.  A  comment  field  is  provided  for  each  item.  Please  provide  any  additional  information  you 
can  to  explain  your  response.  We  encourage  you  to  complete  this  section  since  it  will 
provide  valuable  insights  into  the  usability  of  the  system. 

4.  If  you  have  any  questions,  or  do  not  understand  an  item,  please  contact  the  usability 
evaluation  administrator. 


The  information  you  provide  will  be  anonymous  and  kept  strictly  confidential. 

Thank  you  for  your  participation. 
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1 .  What  is  your  current  j  ob  title?  _ _ 

2.  What  location/facility  are  you  currently  assigned  to?  _ 

3.  What  equipment/systems  are  you  certified  to  maintain?  _ 

4.  How  much  experience  do  you  have  as  an  AF  Specialist?  Years: _  Months: 

5.  How  much  experience  do  you  have  using  NIMS?  Weeks: _  Days:  _ 

6.  Have  you  participated  in  previous  NIMS  evaluations?  Yes _  No _ 

If  so,  when?  Date: _  Which  Version? _ 

7.  What  do  you  feel  is  your  computer  experience  level? 

beginner _  intermediate _  advanced _  expert 

8.  Have  you  received  training  on  NIMS?  Yes _  No _  If  so,  when?  Date: _ 

What  type  of  training  did  you  receive?  Please  check  each  that  applies. 

Classroom _  Hands  on _  Other  (please  specify): _ _ _ 

9.  Is  there  anything  else  we  should  know  regarding  your  participation  in  this  assessment?  _ 
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I.  Visual  Displays 


1 .  Coding  is  used  effectively  (e.g..,  color,  highlighting,  bolding) 

2.  Display  clutter  is  not  a  problem  (i.e.,  information  that  is  presented 
is  necessary  and  well  organized). 

3.  The  meaning  of  each  label  is  immediately  apparent. 

4.  Labels  have  the  same  meanings  as  current  systems. 

5.  Information  that  you  need  is  readily  available  and  well  organized. 

6.  The  design  provides  all  the  necessary  information  for  a  specific 
task  when  it  is  needed  and  in  the  appropriate  sequence. 

7.  The  design  minimizes  visual  attention  to  display  locations  that  will 
interfere  with  attention  to  high  priority  tasks. 

8.  The  design  promotes  situational  awareness  and  service  status 
information. 

9.  Text  is  clear  and  easy  to  read. 

10.  Visual  alerts  are  easy  to  see  and  understand. 

1 1 .  The  computer  responds  quickly  and  you  are  not  kept  waiting  for 
information. 

12. 1  was  always  aware  of  what  the  system  was  doing. 

13.  After  system  recovery  from  degradation  or  failure,  a  smooth  return 
to  operation  is  possible. 


II,  Data  Entry 


14.  The  system  provides  guidance  regarding  data  input  restrictions. 

1 5.  The  data  entry  method(s)  helps  to  minimize  errors. 

16.  It  is  easy  to  detect  errors  in  data  entry. 

17.  It  is  easy  to  correct  errors  in  data  entry. 

1 8.  The  number  of  keystrokes  (or  other  control  actions)  necessary  to 
input  data  is  kept  to  a  minimum. 

19.  The  amount  and  complexity  of  data  entry  is  about  the  same  as  what 
you  are  used  to. 

20.  The  menus  are  easy  to  use  and  do  not  provide  too  many 
options/levels. 

21 .  The  menu  options  are  easy  to  understand. 

22.  It  is  easy  to  know  where  to  find  command. 

23.  Menu  options  are  arranged  in  a  logical  order  with  the  most 
frequently  performed  functions  at  the  top. 
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IF!.  Error  Messages  and  User  Guidance 


24.  Error  messages  are  clear  and  easy  to  understand. 

25.  Each  error  message  briefly  summarizes  the  specific  problem  and 
proposes  a  specific  solution. 

26.  On-line  help  is  easily  accessed. 

27.  On-line  help  is  easy  to  use  and  up-to-date. 


IV.  Data  Input  and  Control  Devices 


28.  The  input  device(s)  is/are  appropriate  for  performing  the  necessary 
functions  (e.g.,  alphanumeric  data  entry,  use  of  windows,  cursor 
positioning). 

29.  The  keyboard  is  easy  to  use. 

30.  The  mouse  is  easy  to  use. 

31.  Function  keys  are  provided  for  frequently  used  commands. 


V.  Functionality 


32.  The  system  design  supports  mobile  users  in  disconnected  mode. 

33.  The  system  design  supports  your  ability  to  generate  Event  Tickets. 

34.  The  system  design  supports  your  ability  to  monitor  the  status  of 
maintenance  activities  (i.e.,  scheduled  and  unscheduled 
maintenance). 

35.  The  system  design  supports  your  ability  to  monitor  the  status  of 
AT  services. 

36.  The  system  supports  your  ability  to  control  remote  AF  facilities. 

37.  The  system  design  supports  your  ability  to  effectively  manage 
resources  (i.e.,  identify  certified  personnel,  assign  personnel,  and 
locate  equipment). 

38.  The  system  design  promotes  your  ability  to  communicate 
effectively  (i.e.,  cc:Mail,  phone,  or  beeper). 

39.  The  system  design  promotes  your  ability  to  coordinate  effectively 
(i.e  contact  field/ AT  personnel,  coordinate  support,  obtain 
approval). 

40.  The  system  design  supports  your  ability  to  generate  reports. 

41.  The  system  provides  electronic  access  to  documents  (e.g.,  facility 
layout  drawings,  and  technical  specifications). 

42.  The  system  provides  a  geographic  display  (i.e.,  map)  with  accurate 
service  status  information. 

43.  The  amount  of  available  functionality  is  adequate. 
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